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ABSTRACT 

This paper provides insights in the development of a skill matching test which addresses soft skills integrated videos as 
media to provide information about situations to be rated. The design of the skill testing and matching tool is situated in 
the educational ERASMUS+ project SMART which is presented as well. With a specific view on team work and the 
necessary skills, traits and interests this article provides insights into the representation of these aspects in the test and 
offers impression of the video and media design. These topics are combined with a presentation of the results of a 
qualitative study concerning this testing tool, which was conducted by expert interviews and analysed by using content 
analysis. These results highlight the advantages and challenges in the use of the testing tool. 
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1. MEDIA TO FOSTER SKILL MATCHING ON THE LABOUR MARKET 

Youth unemployment is an economic and social concern for almost every country affected by the economic 
crisis of the last decade. In Europe the unemployment rate rose from 14.7% in 2008 to 22.2% in 2014. While 
some countries, such as Germany, were able to maintain their unemployment rate (6.7% in March 2017), 
countries like Greece and Spain are suffering from youth unemployment of about 48% and 40.5% (ct. Statista 
2017). In addition to that, nowadays many employers report difficulties in finding suitably skilled workers 
which often ends in a qualification mismatch (ct. GAC on Employment 2014). Qualification mismatch occurs 
when a candidate’s qualification level is higher or lower than that required by the job position (ct. GAC on 
Employment 2014). Moreover, some job positions remain vacant whilst other jobs vanish. Due to 
digitisation, certain jobs are replaced by machines or parts of job tasks are taken over by machines, still those 
machines have to be operated (ct. Frey/ Osborne 2017: 257). Therefore, many job descriptions have changed 
over the years. New job types entered the labour market and new skills have to be developed to serve those 
needs (concerning labour market institutions, requirements and innovation see Bassanini / Ernst 2002). 
Another factor is the societies view on certain job positions which influence the decision of youth when 
applying for a job. Certain jobs have the tendency to be seen as ‘unattractive’ and are not included when 
deciding on a career. Concerning the development of job positions, companies often tend to set requirements 
which are too high and not necessarily needed to fulfil the tasks. These circumstances indicate the need of a 
regulated matching process for the applicants as well as for the companies by providing a framework for 
displaying the skills which are possessed by the applicant as well as which are really needed for a job 
position. Nowadays, many companies offer job positions via internet either through platforms or their own 
website (ct. Pommerien 2011: 53). Furthermore, they prefer applications for jobs in a digital form (ct. ibid). 
This allows companies to search for applicants worldwide. Applicants benefit from using the online 
application, because they are able to choose from a wider range of companies and job positions as well as 
cities or even countries. Hence, using digital media to find a job has become a daily routine. Regarding the 
youth unemployment rate, especially young people with low human capital and who are in the transitional 
system are at risk becoming unemployed (ct. Maxwell 2006: 9). Moreover, if living in rural areas it is even 
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harder to find a suitable job position due to the fact that less job positions might be available. And some 
young people are unable to identity the skillset which they need to apply for a job. Skill tests are not 
uncommon (see e.g. Mind tools 2017 or Assessment Center HQ 2017) and are also available online, but a 
matching process is still missing. Providing a skill testing tool which also manages to match those skills to a 
vacant job position could be very helpful regarding youth, especially in rural areas. The importance of 
providing such a matching process via an online platform is undeniable, since a system can provide such 
algorithms more easily than people. Also a higher number of young learners can be reached via digital media 
and provided with the matching process. Addressing challenges is one of the objectives of the Europe 2020 
strategy, the growth and employment strategy which is implemented by the European Commission during the 
last decade (ct. Erasmus+ Programme Guide: 14). The aim of the Europe 2020 strategy is to overcome the 
economic and social crisis and challenges of past years with intelligent, sustainable and inclusive growth. 
Erasmus+ is the European program to achieve these goals, especially in the context of education, training, 
youth and sport since the year 2014 up to 2020. Within the framework of this program special attention is 
paid to recognizing skills and qualifications as well as promoting justice and integration (ct. Erasmus+ 
Programme Guide: 5). These aspects incline that it is inevitable to help young people develop their human 
capital and become aware of their skills and qualification in order to find employment. The project SMART 
(see Beutner 2016) focuses on youth with school difficulties, economic obstacles or specific needs. 


2. THE SMART PROJECT 

SMART stands for ‘Skill Matching Assistance and Reporting Tool’ and is a ERASMUS+ funded project in 
the key action ‘Cooperation for innovation and the exchange of good practices’ which is embedded in the 
action ‘Strategic Partnerships for youth’ (see Beutner 2016). Therefore, the project is built with an 
educational background and focusses on the analysis of competences and skills. It is founded by the EU, 
started in April 2016 and will run until the end of March 2018. The project idea and structure of SMART was 
developed at the Department of Economic Education II of the University of Paderborn. Project coordinator is 
Asociatia Institutio Pro Educationem Transilvaniensis in Romania, who is characterised by its direct access 
to the target group of SMART, which is youth. Besides these two, three additional partners are assigned to 
the SMART project: University of Dundee in Scotland, Youth for Exchange and Understanding in Cyprus as 
well as Ingenious Knowledge who is responsible for the technical development of the tool. In the following 
the aims, structure and challenges of SMART as well as the SMART Matching Concept will be presented. 

2.1 Aims, Challenges and Structure of Smart 

The main aim of SMART is the development of a matching system which is able to keep pace with the 
changes in society and the labour market to support youth in comparing their competences with the 
requirements of the market (see Beutner 2017). The elements of the tool offer the possibility to create a very 
specific profile of the individuals. The idea is to recognize non-formally acquired skills and competences and 
therefore will be particularly usefi.il for young, disadvantaged youth in the transnational systems who have to 
face different obstacles and who are looking for employment. Thus, SMART enables disadvantaged youth, 
who often lack formal qualifications, to show their real strengths and to present themselves in the most 
beneficial way. In addition to the identification of competences, traits, skills, interests as well as 
qualifications, the tool offers the possibility to match the individual profiles with open vacancies from 
companies, which can be contacted immediately by the user or vice versa (see Beutner 2016). The young 
people using the SMART tool can decide what information to share with potential employers. Due to the fact 
that SMART is developed on a European level the offer of job vacancies will be throughout Europe which 
provides a certain mobility for the youth. Furthermore, youth educators and youth workers can use the 
SMART tool to give young people the opportunity to gain a better understanding of themselves, their 
competences, traits, skills, interest and qualifications (see Beutner 2016 and Beutner 2017). Supporting high 
quality youth work is a necessity to offer occupational orientation. Developing such a tool with the aim to 
deliver a high quality product also entails certain challenges which need to be taken into consideration 
(Beutner 2016). One challenge is a reliable testing (see Golafshani 2003). Without that the tool would lack 
authenticity. The tool should provide fitting information not only to the young learners, but also to the 
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companies. If the tool matches a non-fitting candidate to a job position or the other way around both parts 
will be disappointed and this needs to be avoided. Therefore, the available job position profiles have to be 
specific and reliable, too. Hence, the project partners conducted a sorrow research regarding the demanded 
skills of companies for low skilled job positions as well as identifying the most suitable method to conduct 
the skill testing. Further, regarding the development of the tool the challenge is to design a working matching 
concept. The tool demands a system which is able to produce a result by using the algorithms provided. 
Therefore, it is important how the skills are tested and what the results are and how the job positions are 
measured, because the system needs to connect these results to match the skills to a job profile and vice versa 
(concerning other ways of measuring person-job fit with profiles see Caldwell / O'Reilly 1990). Which 
means the system needs to be programmed in the demanded way as well as the design and output regarding 
the users has to be considered. In fact, the usability of this tool has to be taken into account. Therefore, whilst 
developing the tool, trial runs will give insight in possible improvements. Solving these challenges is crucial 
for providing a high quality tool for all target groups. The structure of the SMART tool entails two parts: (a) 
company related and (b) youth related. First, the system provides the companies with a framework for 
entering their company details. The next step is to provide information about open job positions which they 
want to occupy. Again, the system provides a framework for developing such a profile for open job positions 
in their companies. To ensure that the demands regarding those positions are set in proportion to the tasks the 
employee has to fulfil, the framework asks which skillsets are needed for the position and further asks for the 
priorities regarding those skills which is the first step towards the matching concept. The young applicants 
have to enter personal details guided by a framework at first. This is followed by the testing of their skills. 
Therefore, the young people have to rate certain reactions regarding a work situation presented to them per 
short videos. One video scenario tests different kinds of skills. Behind each rating will be a score. All scores 
regarding one skill will be added and represent the performance of the youth. This will be done for each and 
every skill. After this process the system has a profile regarding the skills of the young person. Those skills 
will be set in relation to each other and provided to the user in a positive way. This happens by not showing 
them their rates in numbers but the relation to each other. So, learners can recognise his or her strengths and 
weaknesses. Just numbers cannot provide any idea of improvements which are maybe needed, expected or 
which the user may wish to undertake. Due to this fact the young people get an overview of their skills, but 
without the judgement of numbers. So, a low skilled person won’t be discouraged by only low numbers, 
because naturally one skill is more developed than another. Now, the youth will be matched with all available 
job positions and will be presented with the results. If the youth want to apply for one of those positions the 
information will be transferred to the companies. Each company will be presented with those candidates that 
match most with the offered position and can decide who they want to interview for the position. 

2.2 The Smart Matching Concept 

Besides providing an easy to understand user interface, the matching process is important. Matching the 
skills of the youth with the job profiles of the companies is, next to the testing, a main part of the SMART 
tool. The testing of the user skills is offered within different working situations through videos. After 
watching a video, the user gets different statements. The task is to rate on a scale how much he or she agrees 
with the single statements. Behind those statements lie skills. Each statement is designed to test a specific 
skill. By setting the scale the system gets the information how to measure this decision. This way it is harder 
to trick the system in answering in a way the user thinks he or she is supposed to. The user sees a rating scale 
next to the statement and has to rank the own opinions. Therefore, the scale is divided in different parts, 
which represent specific skills. Behind these skill parts lie numbers which the system adds up to a final score 
after all statements are rated. The same procedure is being applied for the job profile, only that the companies 
do not have to answer specific statements. The companies need to choose from a range of skills which are 
needed for this specific job position and as a final step have to rate those skills. Now, the system has a score 
for each skill of the user as well as for the job profile. At this moment a matching process is possible. At first, 
the system compares the score of the user skill with the company job profile score. For example, a user 
scored 86 in skilli and the company job profile scored 80 in skilli the system knows that this is a match, 
because as soon as a skill of a user reaches the same score or higher the system marks it as a match. If a skill,, 
score of a user, in this example 63, is not reaching the companies job profile score, in this example 65, it is 
not a match. Depending on the number of matches a user generates when compared to a specific job profile it 
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is a match. For each user the system compares the skill scores with each and every job profile that is 
available. After matching the skills with the job profiles available the system presents the user with all 
matches found. Now, the user can view the company’s profiles and the specific job descriptions and apply for 
the most attractive job positions. The companies will be provided with a set of matching candidates available 
in the SMART tool. The most fitting candidate will be on top of the list followed by lesser fitting candidates, 
but still fitting ones. This way the company can check when receiving an application how fitting the 
candidate is and it can invite the most fitting candidates to an interview. 


3. IDEO DESIGN FOR TESTING SOFT SKILLS 

In SMART tests are crucial to provide enterprises and future employees with a basis for the matching of the 
qualifications of the applicant for a job and the requirements of the business. One important aspect is the 
analysis of soft skills, because they usually cannot be focused as easily as task related or subject oriented 
skills. But test designs which only refer to self-reflection and own estimates often lack a realistic background 
and objectivity. Therefore, the SMART research team decided to provide the job applicants with realistic 
situations and force them to make decisions which show their behavior and offer the chance to get ratings 
about their individual skills. Such rating can be used in two directions. First, they can be used to do the 
matching with a requirement profile provided by the enterprise and for the second, they can be used to 
provide feedback to the applicant about his or her own skill profile as a basis for future development 
strategies. The main challenge is to provide the applicant always with the same situations and to make the 
situations fit to different areas of soft skills. To handle the challenge of providing always the same situation 
to applicants, videos where created to assure comparability and a realistic setting. The use of videos offers 
the advantage to avoid long text descriptions which are not easy to interpret and often boring to the user. 

3.1 Soft Skill Identification and Indication 

One mayor aspect of soft skill which is often addressed by enterprises and in job descriptions is teamwork 
(see as well Robles 2012). But, usually this is not specified in job offers. This fact is challenging for both 
sides for the employer and the applicant. An employer has the problem that it is not easy to select persons 
which fit best, when it is not clear what criteria are really needed. And for the applicant the requirements are 
not transparent. Therefore, it was important in SMART to create a model to measure team competence. The 
team agreed on subskills, traits and interests which should specify this general competence (for a critical 
view on traits see Pervin 1994). Concerning skills, the project partners decided to focus on flexibility and 
communication skills. Herbst/ Guse 2017 defined three sub criteria to explain flexibility: 

1. Willingness to work on short notice 

2. Ability to change from one exercise to another 

3. Willingness to try something new 

And with regard to communication skills they decided to focus on 

1. Ability to understand and express own and other intentions 

2. Being able of suitable and adjusted expression 

3. Willingness to give response/ feedback and to ask questions 

Needed traits are emotional stability and conscientiousness. For emotional stability Herbst/ Guse offered: 

1. Self confidence 

2. Ability to control the own feelings 

3. Ability to cope with stressful situations 
Conscientiousness was addressed by 

1. Sense of Responsibility 

2. Neatness/ Thoroughness 

3. Willingness to achieve effective performance 

Concerning interests, the team differentiated between interest in traditional structured work and realistic 
interest. Traditional interest can be described by the following three criteria: 
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1. Favour material and financial possession 

2. Favour clear instructions 

3. Favour traditional roles 
In contrast, realistic interests are: 

1. Interest in physical work 

2. Down-to-earth-mentality 

3. Prefer concrete over abstract conditions 

3.2 Example of a Video Scenario 

An example how this video and scoring design can look like is the most appropriate way to provide an 
insight into the structure and testing approach at this point. Each video testing scenario is created on the basis 
of a short description which offers five core essentials about it and provides a short overview about the 
structure. They offer an insight in both, the video and the testing elements. 


Video setting 

this section shows core information about the 
environmentand the persons presented in the video 


Possible Reactions 


this part provides text which can be selected by the 
user of the test to answer to the monologue or 
dialogue according to the situation presented in the 
L setting 


Dialogue / Monologue 


this part presents the text 
of the audio which occures 
in the video 


Number and name of the 


Scoring and testing information 


this section of the video scenario offers information 
which main skills and which sub skills are addressed 
with a reaction and part of the scale representing a 

high skill J 


Figure 1. Essential elements of a video scenario description in SMART 

These five essential elements are the basis for the design of each video and scoring part in the whole test. 
They show the connection between what the user hears and sees in the video, the user reactions on it and the 
scoring and skill identification. 

To provide adequate feedback it is important to know which item presented in the possible reactions 
focusses on which main skill. Moreover, a main skill is becoming more concrete by being divided into sub 
skills. But, due to the fact that in a specific behaviour a sub skill cannot be seen directly, each item 
description, which represents a possible reaction, has to be connected to one sub skill. Like in reality it is 
possible to address several sub skills in a video. Therefore, the user has not to select between different 
reactions but has to score how much he or she agrees with this possible reaction. Here a slider will make 
scoring easier. The scale provided has always a high part, a middle part and a low part and it belongs to the 
way in which a possible reaction / the item is described where on the scale high skills are represented. This is 
important to know because in the aggregation process of the different answers to items related to a specific 
sub skill and a skill the rating of the users has to be transferred to single scale with one direction. 


4. QUALITATIVE EXPERT STUDY CONCERNING THE VIDEO 
STRUCTURE 

4.1 Methodology 

After creating the description tables, it was important to provide the results to randomly selected experts to 
get an expert rating about the fit before the making of the video. To gather adequate feedback, the SMART 
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project selected European educational experts. There were always two from UK, Germany, and Romania. 
The first set of six interviews were conducted in March 2017 within three weeks. The second set of 
interviews with another nine interviews was conducted at the end of May 2017 within one week. We did two 
sets because of some changes in the political situation due to elections and the focus on Education in 
Germany and to avoid that this will influence the results. But comparing the results of the first and the second 
set there was no structural or content related difference in the answers of the experts. So, it was possible to 
put both sets of interviews together. Therefore, we are able to derive our result from 15 expert interviews. 

The average duration of each skype or telephone interview (see to the discussion on telephone interviews 
Novick 2008) was about 12 minutes. The interviewer conducted the research after first informative telephone 
contacts. These first information calls ensured that the experts are informed and available. The experts had a 
closer look at the description and the skill model and offered feedback in qualitative interviews. The 
interviews were conducted as telephone and skype interviews in a half-structured way (see Schnell / Hill / 
Esser 1999). Each interview opened with a narrative part, were the expert was able to express the most 
important aspects of the feedback. On the basis of an interview guideline it was assured that specific topics 
were always addressed in each interview: (a) adequacy of the video approach and the video length, (b) 
Connection between possible reactions, sub skills and skills, (c) scoring approach, (d) general feedback on 
the matching process and (e) adequacy of provided settings and dialogues. 

The respondents had always the opportunity to provide additional information. Moreover, there were 
always several parts during the interviews where they could talk in a free not guided way about the topic (see 
Flick 1998; Strauss / Corbin 1998). All experts had the same material and the same introduction. They had at 
least one week before the interview to check the materials provided. In addition to that they got a short 
explanation why the SMART project is focusing on matching of qualifications and skills with the 
requirements of the labour market. They also got an overview about the SMART project. The skype and 
telephone interviews were immediately documented. This documentation process was done via 
text-recordings and structured via argumentation tables. It is important to compare and interpret the feedback 
and to categorise the answers to get deeper insight in a structured way. We used the method of content 
analysis (see Bos / Tarnai 1999) to achieve this aim and to analyse the data. In order to assure 
trustworthiness, the interviews were conducted by one interviewer, who could make sure that every expert 
had the same information and a similar interview structure. The interviews were designed to ensure 
credibility and validity. Validity is a core part of our study and could be realised by gathering all feedback 
and take everything into account. Therefore, all categories presented in the following results emerged from 
the data of the interviews. But, more important is that the categories and feedback components are also 
consistent with the understandings of the experts. For each expert a documentation file was created and 
named EX1 to EX15, which compiles direct quotes and the argumentation tables. These files are used in the 
content analysis. 

4.2 Testing Results and Feedback 

In total the experts stated that the video scenarios are an excellent way to gather the relevant information. 
Also, they agreed that the length of the video has to be very short to ensure that the whole test can be filled in 
with an appropriate time and does not get boring or too long. Concerning (a) adequacy of the video approach 
all 15 experts emphasised that a test on the basis of videos is a really adequate way to solve the task and to 
test the soft skills. One expert was really happy about the approach but added that not only soft skills should 
be addressed and mentioned that “expertise in the field is also needed” (EX3-P2-12). This could be addressed 
in different ways via video, via text based testing and tasks (see EX3-P2-16). The expert recognized that this 
is not the focus of SMART but stressed that this is important for employment as well. 

The connection between possible reactions, sub skills and skills (b) was pretty clear for 13 of the experts. 
Three experts mentioned that it is not compulsory team work which is addressed with the skill set of 
SMART. One expert points out that for example communication is more general and not only related to team 
work (see EX2-P1-21). Nevertheless, it was clear to him that communication skills influence team work. 
Therefore, he was able to accept the skill model behind SMART. A second expert focused on the difference 
between interest in traditional structured work and realistic interest. This expert mentioned that also other 
interest could be taken into account like “interest in innovative modern and creative work” (EX3-P2-26). 
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With regard to the scoring approach (c) all experts agreed that the scoring system is an adequate way to 
collect and gather the data (see e.g. EX5-P1-16). 

Concerning (d) the general feedback on the matching process the 12 of the experts agreed that the 
matching process is necessary and that rural areas need specific support in this field (see e.g. EX1-P1-13 or 
EX6-P2-2). Moreover, they stated that it is necessary to provide both, a feedback to the applicant as well as a 
feedback to the enterprise. One expert suggested: “It is not important that an enterprise gets all data of all 
applicants. Maybe you should take into account that only the information of the best fitting results should be 
provided. This could be the best fitting three applicants.” (EX2-P3-5, translation from mother tongue into 
English by the authors of this article). The last expert added that the matching process is not only necessary 
for rural areas but also for the whole labour market with its sub markets (EX3-P2-19). Two experts didn’t 
comment on that. 

The adequacy of provided settings and dialogues (e) was discussed under different aspects. Some experts 
mentioned that the dialogues should me more realistic concerning wording (see e.g. EC2-P3-10). Other 
experts brought up that the description in the setting could be more concrete (see e.g. EX5-P2-12). All 
experts agreed that the settings and dialogues are a good basis for the filming and provide and excellent 
insight in the situations to decide on. All in all, the results can be shown in this category structure: 


Advantages 


Flexible use of the video 


Integration of authentic situations 


Videos are a guideline through the testing process 


Good usability with videos 


Challenges 


Scoring with sub criteria 
is not always easy 


Duration: The duration of the whole test is crucial 
and must not be too long 


The creation process of the test is very long and 
challenging 


Additional hints 


The acceptance of the users has to be checked 


The organization of reactions, sub skills and skills have to be evaluated for a 
longer period of time to get stable data 


Figure 2. Insights in the categories found as results of the expert interviews 


5. CONCLUSION 

In SMART the results were taken into account and the wording is currently checked by the UK SMART 
team. Moreover, additional research perspectives especially in the field of usability (see Sardonick / Brau 
2015) are set up. A usability study using the TAM model (see e.g. Beutner 2016b) will be conducted soon. In 
addition to that a quantitative field research with users will be a further step in SMART. Overall, the results 
are very positive. They encourage the project team to go further on this way of mapping and testing soft 
skills. The research results show that the use of media, especially videos, enriches and simplifies the 
diagnosis of soft skills and offers a possibility to enhance the process of matching on the labour market. 
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